Proper versus Ad-Hoc MDL Principle for Polynomial Regression

The paper deals with the task of polynomial regression, i.e., inducing polynomial that can be used to predict a chosen dependent variable based on the values of independent ones. As in other induction tasks, there is a trade-off between the complexity of the induced polynomial and its predictive error. One of the approaches for searching an optimal trade-off is the Minimal Description Length principle (MDL). In our previous papers on polynomial regression, we proposed an ad-hoc MDL principle. The focus in this paper is on developing a proper encoding schema for polynomials that leads to a proper MDL principle for polynomial regression. We implemented the developed MDL principle as a search heuristic in CIPER, an algorithm for inducing polynomials from data. We present an empirical comparison between the heuristics based on the ad-hoc and the proper MDL principle. The results show that proper MDL principle leads to simpler polynomials with comparable predictive error. Finally, we also propose a lower bound for the proper MDL principle that allows branch-and-bound pruning of the CIPER search space and evaluate the benefits of pruning.